MigratoryData Kafka Edition

Apache Kafka is a popular realtime messaging technology with a rich ecosystem. More than 100 connectors are available for technologies like stream processing, big data, machine learning, databases, and others to get data in and out of Kafka. Therefore, Kafka excels at connecting backend technologies, usually across a trusted and reliable network like Intranet.

Connecting realtime web and mobile apps to Kafka over an untrusted and unreliable network like Internet while preserving the key features of Kafka like guaranteed delivery, message ordering, scalability to handle large volumes of messages, and high availability is far from being simple task.

There are some attempts to access Kafka through the Websocket protocol, which is a good start. But simply using WebSockets doesn’t offer much help as this does not scale. Indeed, Kafka has been designed for connecting a relatively few backend systems. However, web and mobile apps usually have many users. Not being optimized to handle lots of connections and topics, Kafka should offload the web messaging task to another realtime messaging layer.

MigratoryData, being able to handle 10 million concurrent WebSocket connections and 10 million concurrent topics on a single commodity server, is best suited to offload and accelerate Kafka messaging across the web. Moreover, its enterprise messaging features, proven in production for over a decade, are perfectly compatible with Kafka. So, MigratoryData naturally extends to the Internet the Kafka’s key features like ordering and delivery guarantees and high availability, all without putting any pressure on Kafka.